Towards Energy Efficient MapReduce
نویسندگان
چکیده
Energy considerations are important for Internet datacenters operators, and MapReduce is a common Internet datacenter application. In this work, we use the energy efficiency of MapReduce as a new perspective for increasing Internet datacenter productivity. We offer a framework to analyze software energy efficiency in general, and MapReduce energy efficiency in particular. We characterize the performance of the Hadoop implementation of MapReduce under different workloads. We also introduce quantitative models to guide operators and developers in improving the performance of MapReduce/Hadoop. A major, but somewhat unsurprising finding is that for workloads where the work rate is proportional to the amount of resources used, improving the performance as measured by traditional metrics such as job duration is equivalent to improving the performance as measured by lower energy consumed.
منابع مشابه
Statistical Workloads for Energy Efficient MapReduce
Energy efficiency is a growing concern in modern datacenters. As Internet services increasingly rely on MapReduce workloads to fuel their flagship businesses, there is a growing need for better MapReduce energy efficency evaluation mechanisms. We present a statistics-driven workload generation framework that distills summary statistics from production MapReduce traces and realistically reproduc...
متن کاملReducing Cluster Energy Consumption through Workload Management
Energy consumption is a major and costly problem in data centers. For many workloads, a large fraction of energy goes to powering idle machines that are not doing any useful work. There are two causes of this inefficiency: low server utilization and a lack of power proportionality. We focus on addressing this problem for two workloads: (1) a traditional, front-end web server workload and (2) an...
متن کاملTowards Efficient MapReduce Using MPI
MapReduce is an emerging programming paradigm for dataparallel applications. We discuss common strategies to implement a MapReduce runtime and propose an optimized implementation on top of MPI. Our implementation combines redistribution and reduce and moves them into the network. This approach especially benefits applications with a limited number of output keys in the map phase. We also show h...
متن کاملTowards Efficient Large-Scale Feature-Rich Statistical Machine Translation
We present the system we developed to provide efficient large-scale feature-rich discriminative training for machine translation. We describe how we integrate with MapReduce using Hadoop streaming to allow arbitrarily scaling the tuning set and utilizing a sparse feature set. We report our findings on German-English and RussianEnglish translation, and discuss benefits, as well as obstacles, to ...
متن کامل